A Content-Adaptive Analysis and Representation Framework for Audio Event Discovery from "Unscripted" Multimedia

We propose a content-adaptive analysis and representation framework to discover events using audio features from ünscriptedm̈ultimedia such as sports and surveillance for summarization. The proposed analysis framework performs an inlier/outlier based temporal segmentation of the content. It is motivated by the observation that ı̈nterestingëvents in unscripted multimedia occur sparsely in a background of usual or üninterestingëvents. We treat the sequence of low / mid level features extracted from the audio as a time series and identify subsequences that are outliers. The outlier detection is based on eigenvector analysis of the affinity matrix constructed from statistical models estimated from the subsequences of the time series. We define the confidence measure on each of the detected outliers as the probability that it is an outlier. Then, we establish a relationship between the parameters of the proposed framework and the confidence measure. Furthermore, we use the confidence measure to rank the detected outliers in terms of their departures from the background process. Our experimental results with sequences of low and mid level audio features extracted from sports video show that ḧighlightëvents can be extracted effectively as outliers from a background process using the proposed framework. We proceed to show the effectiveness of the proposed framework in bringing out suspicious events from surveillance videos without any a priori knowledge. We show that such temporal segmentation into background and outliers, along with the ranking based on the departure from the background, can be used to generate content summaries of any desired length. Finally, we also show that the proposed framework can be used to systematically select k̈ey audio classesẗhat are indicative of events of interest in the chosen domain. Eurasip Journal of Applied Signal Processing This work may not be copied or reproduced in whole or in part for any commercial purpose. Permission to copy in whole or in part without payment of fee is granted for nonprofit educational and research purposes provided that all such whole or partial copies include the following: a notice that such copying is by permission of Mitsubishi Electric Research Laboratories, Inc.; an acknowledgment of the authors and individual contributions to the work; and all applicable portions of the copyright notice. Copying, reproduction, or republishing for any other purpose shall require a license with payment of fee to Mitsubishi Electric Research Laboratories, Inc. All rights reserved. Copyright c ©Mitsubishi Electric Research Laboratories, Inc., 2005 201 Broadway, Cambridge, Massachusetts 02139